A typical use of a search engine involves a user, a search query and a client device. The user submits the search query through the client device to the search engine. Upon receipt of the search query, the search engine identifies in its database one or more items relevant to the search query according to a set of predefined criteria, and returns the identified items to the client device as a search result (e.g., documents, advertisements and/or units of information such as dictionary definitions). The items are ordered based upon their respective relevancy or ranking values according to a predefined empirical standard (e.g., PageRank, as described in detail in U.S. Pat. No. 6,285,999, hereby incorporated by reference in its entirety). An item with a higher ranking value usually appears before those of lower ones.
The click through rate of an informational item may be defined as the ratio of the number of times the item has been selected by a user (sometimes called the number of click throughs) to the number of times the item was displayed (sometimes called the number of impressions), and thereby made available for selection. In at least one widely used system, advertisements corresponding to displayed content are ordered in accordance with an ordering function, one parameter of which is the click through rates of the items (advertisements) being ordered.
In at least some situations, when displaying a set of informational items, informational items having high click through rates should be ranked higher, and displayed more prominently than informational items with lower click through rates, because the items with high click through rates are more likely to be of interest to the user(s).
However, if the information being sought, e.g., the meaning of a word, is already available from an informational item as presented in the search results (i.e., not requiring the user to click though to receive additional information), the search engine may not receive user feedback about the relevancy of the informational item to the search. The user has little motivation to provide feedback such as a click through because the user now has the information sought, or there may be no outbound link associated with the information item. Therefore, there is a need for collecting user feedback with respect to informational items presented in response to a search query and adjusting the relevancy values of the items in accordance with the user feedback, even if there is no click through information about one or more of the informational items.